Method and system for providing mixed reality service

ABSTRACT

The present invention relates to a method for providing a mixed reality service by a server and a system for performing the same, the method comprising the steps of: generating a mixed reality conference room for arbitrary three-dimensional modeling data; receiving tracking information collected by at least one terminal participating in the mixed reality conference room; generating a virtual image for the three-dimensional modeling data and at least one object, on the basis of the tracking information; and transmitting the virtual image to the at least one terminal, wherein the tracking information includes space identification information of a reality space of each terminal and motion information of each terminal, and the virtual image is synthesized with a reality image in the at least one terminal so as to be displayed as a mixed reality image.

TECHNICAL FIELD

The present invention relates to a mixed reality service providingmethod and system, and more specifically, to a mixed reality serviceproviding method and system which allow a plurality of terminal users toshare three-dimensional modeling data in a mixed reality space.

BACKGROUND ART

Recently, with the rapid development of three-dimensional imageprocessing technology, services using virtual reality (VR) or augmentedreality (AR) are being provided in various fields such as movies, games,and interiors. Mixed reality (MR), which appeared after AR, synthesizesand displays a reality image with a virtual image implemented usingthree-dimensional modeling data and thus provides an image without aboundary between the reality image and the virtual image to a user.

Meanwhile, in the field of building design, technologies, in whichdesign data may be made into three-dimensional modeling data so thatstructures of buildings are recognized more three-dimensionally, arealso being developed.

In addition to building companies, various service companies forbuilding interior facilities and interiors, such as electricity and/orgas facilities, participate in the design of buildings andsimultaneously perform a process of pieces of design data. In theconventional sharing method, there is a problem in that the process ofdesign data is not fully completed or is mishandled, resulting ininefficient building design.

In the current trend of rapidly developing building design andtechnology and rapidly sharing information by technicians in variousfields, providing a service that allows remote users to share data inreal time and collaborate is in demand.

DISCLOSURE Technical Problem

The present invention is directed to providing a mixed reality serviceproviding method and system which allow a plurality of users to sharethree-dimensional modeling data in real time and collaborate remotely ina mixed reality space.

The present invention is also directed to providing a mixed realityservice providing method and system in which, in a mixed reality space,three-dimensional modeling data and various objects are scaled anddisplayed to be appropriate for a reality space of a user.

Technical Solution

One aspect of the present invention provides a mixed reality serviceproviding method, which is a method of providing a mixed reality serviceof a server. The method includes generating a mixed reality conferenceroom for arbitrary three-dimensional modeling data, receiving trackinginformation collected by at least one terminal that attends the mixedreality conference room, generating virtual images of thethree-dimensional modeling data and at least one object on the basis ofthe tracking information, and transmitting the virtual images to the atleast one terminal. The tracking information includes spaceidentification information about a reality space of each terminal andmovement information of each terminal, and the virtual images aresynthesized with a reality image in the at least one terminal, and thesynthesized image is displayed as a mixed reality image.

The generating of the virtual images may include determining at leastone of a display direction and a shape of the three-dimensional modelingdata on the basis of movement information of each terminal, determininga planar region from the space identification information of eachterminal, and arranging the three-dimensional modeling data in theplanar region.

The generating of the virtual images may further include enlarging orreducing a size of the three-dimensional modeling data according to asize of the planar region.

The generating of the virtual images may include generating at least oneobject that imitates a user of each terminal, and determining at leastone of a display direction and a position of the at least one object onthe basis of movement information of each terminal.

The determining of the at least one of the display direction and theposition of the at least one object may further include determining anarbitrary reference point from the space identification information ofeach terminal, determining a size of a reality space of each terminalfrom the space identification information of each terminal, andarranging the at least one object in a concentrated or distributedmanner based on the arbitrary reference point according to the size ofthe reality space of each terminal.

The method may further include receiving a request for processing of thethree-dimensional modeling data through at least one user interfacedisplayed on the mixed reality image, editing the three-dimensionalmodeling data in response to the request for the processing,regenerating the virtual images on the basis of the editedthree-dimensional modeling data, and transmitting the regeneratedvirtual images to the at least one terminal.

The three-dimensional modeling data may be any one of modeling data forbuilding design, three-dimensional data for a product, andthree-dimensional modeling data generated in a design stage.

The method may further include transmitting the mixed reality imagedisplayed in the terminal to a monitoring device so as to be displayedon the monitoring device for at least one user other than a user of theterminal.

Another aspect of the present invention provides a mixed reality serviceproviding system, which is a system for providing a mixed realityservice. The system includes at least one terminal configured to collectand transmit tracking information while attending a mixed realityconference room, and a server configured to generate a mixed realityconference room for arbitrary three-dimensional modeling data, generatevirtual images of the three-dimensional modeling data and at least oneobject on the basis of the tracking information received from the atleast one terminal that attends the mixed reality conference room, andtransmit the generated virtual images to the at least one terminal. Thetracking information includes space identification information about areality space of each terminal and movement information of eachterminal, and the at least one terminal synthesizes a reality imagecaptured by the at least one terminal with the virtual images anddisplays the synthesized image as a mixed reality image.

The server may determine at least one of a display direction and a shapeof the three-dimensional modeling data on the basis of movementinformation of each terminal, determine a planar region from the spaceidentification information of each terminal, and arrange thethree-dimensional modeling data in the planar region to generate thevirtual image.

The server may enlarge or reduce a size of the three-dimensionalmodeling data according to a size of the planar region and generate thevirtual image.

The server may generate at least one object that imitates a user of eachterminal, determine at least one of a display direction and a positionof the at least one object on the basis of movement information of eachterminal, and generate the virtual image.

The server may determine an arbitrary reference point from the spaceidentification information of each terminal, determine a size of areality space of each terminal from the space identification informationof each terminal, arrange the at least one object in a concentrated ordistributed manner based on the arbitrary reference point according tothe size of the reality space of each terminal, and generate the virtualimage.

The terminal may transmit a request for processing of thethree-dimensional modeling data to the server on the basis of a userinput received through at least one user interface displayed on themixed reality image, and the server may edit the three-dimensionalmodeling data in response to the request for the processing, regeneratethe virtual images on the basis of the edited three-dimensional modelingdata, and transmit the regenerated virtual images to the at least oneterminal.

The three-dimensional modeling data may be any one of modeling data forbuilding design, three-dimensional data for a product, andthree-dimensional modeling data generated in a design stage.

The system may further include a monitoring device configured to receivethe mixed reality image displayed in the terminal from the terminal anddisplay the mixed reality image for at least one user other than a userof the terminal.

Advantageous Effects

According to a mixed reality service providing method and systemaccording to the present invention, in a mixed reality space, remoteusers can realistically view three-dimensional modeling data in realtime and share an editing status and thus enable more efficientcollaboration.

Further, according to the mixed reality service providing method andsystem according to the present invention, in the field of buildingdesign, it is possible to maximize the use of three-dimensional modelingdata and minimize the time and resources required for building design.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating a network structure of a mixed realityservice providing system according to the present invention.

FIG. 2 is a diagram illustrating a configuration of a server accordingto the present invention.

FIG. 3 is a diagram illustrating a configuration of a terminal accordingto the present invention.

FIG. 4 is a diagram illustrating an example in which a terminalaccording to the present invention is implemented as a head-mounteddisplay device.

FIG. 5 is a flowchart illustrating a method of driving a mixed realityservice according to the present invention.

FIG. 6 is a flowchart illustrating a method of storing three-dimensionalmodeling data according to the present invention.

FIGS. 7 and 8 are flowcharts illustrating a method of providing a mixedreality conference room according to the present invention.

FIGS. 9 and 10 are diagrams for describing three-dimensional scalingaccording to the present invention.

FIG. 11 is a diagram illustrating an example of a mixed reality imageaccording to the present invention.

FIG. 12 is a diagram illustrating a network structure of a mixed realityservice providing system according to additional embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE INVENTION

In the description of embodiments in this specification, when it isdetermined that detailed descriptions of related well-knownconfigurations or functions unnecessarily obscure the gist of thepresent invention, detailed descriptions thereof will be omitted.

The expressions “includes,” “can include,” etc. used herein indicate thepresence of the disclosed corresponding functions, operations, elements,and the like, and do not limit one or more additional functions,operations, elements, and the like. In addition, it should be furtherunderstood that the terms “comprise,” “comprising,” “include,” and/or“including,” when used herein, specify the presence of stated features,integers, steps, operations, elements, parts, or combinations thereofbut do not preclude the presence or addition of one or more otherfeatures, integers, steps, operations, elements, parts, or combinationsthereof.

As used herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise.

A mixed reality service providing method and system according to thepresent invention may be provided for building design. Morespecifically, the mixed reality service providing method and systemaccording to the present invention may be configured to display designdata of a building, which serves as three-dimensional modeling data, ona plurality of terminals of users entering a mixed reality conferenceroom as mixed reality and to manage (add, modify, delete, or the like)design information.

Hereinafter, the present invention will be described with reference tothe accompanying drawings.

FIG. 1 is a diagram illustrating a network structure of a mixed realityservice providing system according to the present invention.

Referring to FIG. 1, a mixed reality service providing system 1according to the present invention may include a server 10 that providesa mixed reality service and at least one terminal 20 to which the mixedreality service is provided.

The server 10 is provided in order to provide the mixed reality serviceaccording to the present invention and may be a network server, anapplication server, a domain server, or the like operated by a providerof the mixed reality service.

When the server 10 receives a request for providing of a mixed realityservice from the terminal 20, the server 10 opens a mixed realityconference room and provides a virtual image of three-dimensionalmodeling data, which is processed to correspond to a reality space ofthe terminal 20 attending the conference room, to the terminal 20. Thevirtual image provided to the terminal 20 may be synthesized with areality image captured by the terminal 20, and the synthesized image maybe displayed as a mixed reality image.

The terminal 20 performs data communication with the server 10 to beprovided with the mixed reality service.

The terminal 20 may include portable units, such as cellular phones,smart phones, personal digital assistants (PDAs), tablet computers, andpersonal computers. In particular, in various embodiments of the presentinvention, the terminal 20 may be configured as a head-mounted display(HMD), as illustrated in FIG. 4, so that the user may more realisticallyexperience the three-dimensional modeling data.

The server 10 and the terminal 20 are connected to each other via anetwork to perform data communication and perform a control operationfor the mixed reality service. A more detailed description thereof willbe given below.

FIG. 2 is a diagram illustrating a configuration of the server accordingto the present invention.

Referring to FIG. 2, the server 10 may include a communication unit 11,a control unit 12, and a storage unit 13.

The communication unit 11 may transmit and receive data to and from theoutside via a network. For example, the communication unit 11 mayreceive a request corresponding to a user input and information requiredfor generating a virtual image from the terminal 20 and transmit thevirtual image which will be synthesized on a reality image to theterminal 20 under the control of the control unit 12.

In order to provide the mixed reality service according to the presentinvention, the control unit 12 is configured to control each componentof the server 10. More specifically, the control unit 12 may include adata conversion unit 121, a mixed reality service providing unit 122,and a virtual image generation unit 123.

The data conversion unit 121 converts two-dimensional modeling data tothree-dimensions to generate three-dimensional modeling data. When thedata conversion unit 121 receives two-dimensional modeling data from theterminal 20 or the like via the network, the data conversion unit 121may convert the two-dimensional modeling data into three-dimensionalmodeling data using designated software (e.g., AutoCAD developed byAutodesk, Inc., etc.). Alternatively, even when the modeling datareceived from the terminal 20 or the like via the network is thethree-dimensional modeling data, the data conversion unit 121 mayconvert the modeling data into three-dimensional modeling data to have aformat that supports the received three-dimensional modeling data aslong as the format is not a format supported by the server 10 in orderto provide the mixed reality service.

In various embodiments of the present invention, the three-dimensionalmodeling data may be modeling data for building design. Thethree-dimensional modeling data may be three-dimensional data for aproduct, three-dimensional modeling data generated in a design stage,and the like. In particular, in various embodiments of the presentinvention, the three-dimensional modeling data may be IndustryFoundation Classes (IFC) data as building information modeling (BIM)data, but the present invention is not limited thereto. Thethree-dimensional modeling data for building design may includeinformation associated with a structure, a shape, a size, a material, aquality of a material, a color, a pattern, and a facility of thebuilding.

In one embodiment, when the data received from the terminal 20 or thelike is three-dimensional modeling data in a preset format, a separateconversion process by the data conversion unit 121 may not be performed.

The data conversion unit 121 may store and manage the converted orreceived three-dimensional modeling data in the storage unit 13. Invarious embodiments of the present invention, when a user account forthe terminal 20 is generated by the mixed reality service providing unit122, the data conversion unit 121 may store three-dimensional modelingdata of the corresponding user in a data directory designated for eachuser account.

The mixed reality service providing unit 122 performs various types ofcontrol operations for providing the mixed reality service to theterminal 20.

Specifically, the mixed reality service providing unit 122 may generateand manage a user account of the terminal 20 to which the mixed realityservice is provided. The mixed reality service providing unit 122 maystore and manage data, which is generated or obtained through the mixedreality service, in association with the user account. For example, themixed reality service providing unit 122 may store and manage anidentification (ID) and a password as pieces of identificationinformation for identifying the user account and store and managemodeling data, a mixed reality conference room, and the like, which areassociated with the corresponding user account, in a data directorycorresponding to the user account.

When the mixed reality service is driven by a request of the terminal20, the mixed reality service providing unit 122 may receive informationrequired for generating the virtual image, for example, trackinginformation, from the terminal 20, and transmit the received informationto the virtual image generation unit 123. Further, the mixed realityservice providing unit 122 may transmit the virtual image generated bythe virtual image generation unit 123 to the terminal 20 so that thevirtual image may be synthesized with the reality image in the terminal20 to generate a mixed reality image.

While the mixed reality service is provided, the mixed reality serviceproviding unit 122 may edit and store the three-dimensional modelingdata in response to the user input received from the terminal 20. Themixed reality service providing unit 122 may transmit the editedthree-dimensional modeling data to the virtual image generation unit 123so that the virtual image may be generated based on the editedthree-dimensional modeling data. When the three-dimensional modelingdata is edited by an arbitrary terminal 20, the virtual imageregenerated based on the edited three-dimensional modeling data may betransmitted to the corresponding terminal 20 and one or more otherterminals 20. Accordingly, the terminals 20 to which the mixed realityservice is provided may perform collaboration simultaneously whilesharing the editing state of the three-dimensional modeling data.

Further, the mixed reality service providing unit 122 may performfunctions, such as text chatting, voice chatting, and group call, whilethe mixed reality service is provided. Accordingly the mixed realityservice providing unit 122 may transmit text data and/or voice datareceived from the terminal 20 to the one or more other terminals. Invarious embodiments of the present invention, the voice chattingfunction or the group call function may allow only a specific terminal(e.g., a master terminal) to output voice or may be performed by amethod of getting the right to speak (e.g., a push-to-talk method).

The virtual image generation unit 123 may generate the virtual image onthe basis of the tracking information received from the terminal 20. Thetracking information may include, for example, identificationinformation (e.g., mesh network information) about the reality space ofthe terminal 20, three-dimensional rotation information, movementinformation, and the like of the terminal 20.

The virtual image generation unit 123 may generate the virtual image byrotating or editing the three-dimensional modeling data on the basis ofthe tracking information of the terminal 20, which is received from theterminal 20, or arranging the three-dimensional modeling data at aspecific position. Alternatively, the virtual image generation unit 123may generate the virtual image by rotating a user interface object onthe basis of the tracking information of the terminal 20 or arrangingthe three-dimensional modeling data at the specific position.Alternatively, the virtual image generation unit 123 may generate thevirtual image by rotating an object that imitates the user, for example,an avatar, on the basis of the tracking information of the terminal 20or arranging the three-dimensional modeling data at the specificposition.

In various embodiments, the virtual image generation unit 123 maygenerate the virtual image by scaling (enlarging or reducing) thethree-dimensional modeling data on the basis of the tracking informationof the terminal 20. Alternatively, the virtual image generation unit 123may generate the virtual image by scaling (concentrating, distributing,or rearranging) the three-dimensional modeling data based on anarbitrary reference point with respect to at least one object thatimitates at least one user on the basis of the tracking information ofthe terminal 20.

The above-described components of the control unit 12 may be implementedas physically separate devices in the control unit 12, but thetechnological scope of the present invention is not limited thereto.That is, the above-described components of the control unit 12 may bemodularized or programmed in one physical processor. Further, theabove-described components of the control unit 12 are merely dividedinto operational aspects of the control unit 12, and at least one or allof the above-described components may be integrated into one component.

The storage unit 13 may store an operating system, programs, software,and the like required for the operation of the server 10. In variousembodiments of the present invention, the storage unit 13 may store atleast one piece of three-dimensional modeling data and data (e.g., imagedata of the user interface object or the object that imitates the user)required to provide the mixed reality service under the control of thecontrol unit 12.

FIG. 3 is a diagram illustrating a configuration of the terminalaccording to the present invention.

Referring to FIG. 3, the terminal 20 according to the present inventionmay include a camera unit 21, a sensor unit 22, an input unit 23, adisplay unit 24, an output unit 25, a communication unit 26, a controlunit 27, and a storage unit 28.

The camera unit 21 includes at least one camera module to capture animage in front of the user. In various embodiments of the presentinvention, the camera unit 21 may include a depth camera that canidentify a shape and a sense of depth of a reality space (subject) of auser. Further, the camera unit 21 may include an infrared camera or thelike for photographing a hand gesture of the user.

The sensor unit 22 may include at least one sensor that can sensevarious pieces of information about operations of the terminal 20. Invarious embodiments of the present invention, the sensor unit 22 mayinclude a gyro sensor, an acceleration sensor, a Global PositioningSystem (GPS) sensor, and the like for sensing movements such asthree-dimensional rotation, inclination, and movement of the terminal20.

The input unit 23 receives various inputs from the user. The input unit23 may include a touch pad, a keypad, a jog dial, a dome switch, abutton, or the like. Alternatively, in one embodiment, when the inputunit 23 receives a hand gesture of the user through the camera unit 21as an input, the input unit 23 may be configured to perform an operationof identifying the hand gesture photographed by the camera unit 21.Further, in various embodiments, the input unit 23 may further include amicrophone that receives sound, such as a user's voice, and processesthe sound as electrical voice data.

The display unit 24 may visually display various pieces of informationprocessed by the terminal 20. In various embodiments of the presentinvention, when the terminal 20 is provided as an HMD 30, the displayunit 24 may include a display unit for a left eye which displays aleft-eye image, and a display unit for a right eye which displays aright-eye image.

The output unit 25 is configured to output pieces of informationprocessed by the terminal 20 in various forms such as sound, vibration,and light. To this end, the output unit 25 may include a speaker, ahaptic module, a light-emitting diode (LED) lamp, and the like.

The communication unit 26 may transmit and receive data to and from theoutside via a network. For example, the communication unit 26 maytransmit various requests and/or various pieces of information for beingprovided with the mixed reality service to the server 10 and receivethree-dimensional modeling data and/or virtual images about variousobjects from the server 10 under the control of the control unit 27.

In order to be provided with the mixed reality service according to thepresent invention, the control unit 27 is configured to control eachcomponent of the terminal 20. More specifically, the control unit 27 mayinclude a data management unit 271, a mixed reality service managementunit 272, and a mixed reality generation unit 273.

The data management unit 271 may generate modeling data or downloadmodeling data from the outside, such as the server 10, in response to auser input, and manage (i.e., edit, delete, store, etc.) the modelingdata. In various embodiments, the data management unit 271 may drivevarious modelling programs, such as AutoCAD developed by Autodesk, Inc.,ArchiCAD, and the like, to generate, edit, and convert the modelingdata.

The mixed reality service management unit 272 controls operationsassociated with the mixed reality service provided by the server 10.

Specifically, the mixed reality service management unit 272 may transmita request for generation and/or management of a user account or arequest for login with the user account to the server 10 in response tothe user input. The mixed reality service management unit 272 mayreceive an ID, a password, and the like as identification informationfor generating the user account or logging in with the user account fromthe user, and transmit the received ID and the password to the server10.

The mixed reality service management unit 272 may transmit the modelingdata generated by the data management unit 271 to the server 10 ordownload the modeling data from the server 10 in response to the userinput.

The mixed reality service management unit 272 may transmit a request fordriving of the mixed reality service to the server 10 in response to theuser input. When the mixed reality service is driven, the mixed realityservice management unit 272 may collect tracking information using thecamera unit 21 and/or the sensor unit 22 and transmit the collectedtracking information to the server 10. The mixed reality servicemanagement unit 272 may receive a virtual image, which is generatedbased on the tracking information, from the server 10. The mixed realityservice management unit 272 may transmit the received virtual image tothe mixed reality generation unit 273 and synthesize a reality imagecaptured by the camera unit 21 with the virtual image so that a mixedreality image is generated.

While the mixed reality service is provided, the mixed reality servicemanagement unit 272 may receive a user input, for example, an input forediting, or the like of the three-dimensional modeling data, through atleast one user interface which is received from the server 10 anddisplayed by the mixed reality generation unit 273. The mixed realityservice management unit 272 may process the user input to transmit theprocessed user input to the server 10, and receive and process aresponse thereof from the server 10.

While the mixed reality service is provided, the mixed reality servicemanagement unit 272 may receive text or voice through the input unit 23as an input, process text or voice, and transmit the processed text orvoice to the server 10 in order to perform functions, such as textchatting, voice chatting, and group call, which are provided by theserver 10. The input of text or voice transmitted to the server 10 maybe transmitted to other terminals that simultaneously use the mixedreality service through the server 10.

The mixed reality generation unit 273 synthesizes the reality imagecaptured by the camera unit 21 with the virtual image received from theserver 10 to generate the mixed reality image. The generated mixedreality image may be displayed through the display unit 24.

In one embodiment, as will be described below, when the terminal 20 isprovided as a HMD 30, the mixed reality generation unit 273 may generatea left-eye image and a right-eye image of the mixed reality image. Thegenerated left-eye image and right-eye image may be displayed on thedisplay unit for a left eye and the display unit for a right eye of theHMD 30, respectively.

The storage unit 28 may store an operating system, programs, software,and the like required for the operation of the terminal 20.

Meanwhile, in various embodiments of the present invention, the terminal20 may be provided as a HMD 30, as illustrated in FIG. 4.

The HMD 30 may be provided as a frame 29. The frame 29 may be formed ofa flexible material to be easily worn on the user's head and may beformed, for example, in the form of glasses to be worn on the user'sface. The frame 29 may be referred to as a main body portion or a bodyportion. The camera unit 21, the sensor unit 22, the input unit 23, thedisplay unit 24, and the output unit 25, which are described above, maybe provided in the frame 29.

The display unit 24 may include a display unit 24 for a left eye and adisplay unit 24 for a right eye which correspond to a left eye and aright eye, respectively, while being worn by the user. Accordingly, theHMD 30 may allow the user to feel a sense of depth corresponding toparallax between an image for a left eye and an image for a right eyeand experience a more realistic mixed reality space.

However, a structure of the HMD 30 is not limited to the above, and theHMD may have various structures and shapes.

Hereinafter, a mixed reality service providing method will be describedin more detail with reference to the mixed reality service providingsystem 1 according to the present invention. The mixed reality serviceproviding method to be described below may be performed usingapplications, programs, software, or the like installed on the terminal20 or may be performed using a Hypertext Transfer Protocol (HTTP)-basedweb service. However, the technological scope of the present inventionis not limited thereto, and the mixed reality service providing methodaccording to the present invention may be performed in various manners.

FIG. 5 is a flowchart illustrating a method of driving the mixed realityservice according to the present invention.

Referring to FIG. 5, the terminal 20 may receive a user input fordriving a mixed reality service (501). For example, the user input fordriving the mixed reality service may be received by executingapplications, programs, or software for providing the mixed realityservice or by moving to a web page that provides the mixed realityservice.

The terminal 20 may transmit a request for driving of the mixed realityservice to the server 10 in response to the user input (502).

The server 10 may transmit a driving response with respect to thedriving request of the terminal 20 to the terminal 20 (503). In variousembodiments, the server 10 may perform device authentication and/orsecurity authentication on the terminal 20, determine whether to providethe mixed reality service to the terminal 20, and transmit the drivingresponse to the terminal 20.

In various embodiments, the authentication process described above maynot be performed separately.

When the terminal 20 receives the driving response of the mixed realityservice, the terminal 20 may collect tracking information in real time(504). In various embodiments, the terminal 20 may collect the trackinginformation using the camera unit 21 and/or the sensor unit 22.

For example, the terminal 20 may analyze an image of the reality spacecaptured by the camera unit 21 to identify a shape of the reality space,a sense of depth, and the like. Through space identification, theterminal 20 may identify a region of a specific shape, for example, aplanar region on which three-dimensional modeling data may be displayed.Further, through the space identification, the terminal 20 may determinea shape, a size, and a position of an obstacle (e.g., furniture, etc.)located in front of the user.

Further, for example, the terminal 20 may sense movements such asthree-dimensional rotation, inclination, and movement of the terminal 20using the sensor unit 22.

In the present invention, the tracking information collected by theterminal 20 is not limited to the space identification information andthe movement information which are described above and may includevarious pieces of information required for generating the virtual image,for example, marker recognition information, hand gesture identificationinformation, and the like.

The terminal 20 may transmit the collected tracking information to theserver 10 in real time (505).

The server 10 may generate the virtual image on the basis of thereceived tracking information (506). Specifically, the server 10 maygenerate an object required to provide the mixed reality service in theform of a virtual image. For example, the server 10 may generate a userinterface object in the form of a virtual image. The user interface mayinclude a user interface for account generation, logging in with theaccount, account management, upload or download of modeling data, orattending or management of a mixed reality conference room.

The server 10 may rotate or enlarge or reduce an image of the object asan effect corresponding to the tracking information. Further, the server10 may determine a position of the object on a screen displayed on thedisplay unit 24 of the terminal 20.

The server 10 may transmit the generated virtual image to the terminal20 (507).

The terminal 20 may synthesize the received virtual image with thereality image captured by the camera unit 21 and generate a mixedreality image (508). The mixed reality image generated in this way maybe an image in which a two- or three-dimensional image of the userinterface required to use the mixed reality service is arranged in thereality space in which the terminal 20 is located.

The terminal 20 may display the generated mixed reality image throughthe display unit 24 (509).

While the mixed reality image is displayed, the terminal 20 continuouslycollects the tracking information in real time and transmits thecollected tracking information to the server 10. Further, the server 10may generate a virtual image on the basis of the tracking informationthat is continuously received and transmit the virtual image to theterminal 20. The terminal 20 may synthesize the virtual image changedaccording to the tracking information collected in real time with thereality image and display the mixed reality image to the user.

While the tracking information is collected, the terminal 20 may sense auser input, which is input, using the camera unit 21, the sensor unit22, and/or the input unit 23. The user input may be received through thedisplayed user interface. For example, the terminal 20 may sense theuser input for generating the user account for using the mixed realityservice through an account generation user interface. Alternatively, theterminal 20 may sense the user input for logging in with the useraccount already generated through an account login interface.

The user input may be received through the displayed user interface. Invarious embodiments of the present invention, the terminal 20 may sensethe user input by identifying a hand gesture of the user using aninfrared camera. For example, when a hand gesture pointing a specificposition with a finger is identified, the terminal 20 may sense a userinput for executing the user interface displayed at the position pointedby the finger. Alternatively, for example, when a hand gesture foldingand unfolding any finger is identified, the terminal 20 may sense a userinput for displaying a list of mixed reality conference rooms. In thepresent invention, the user input corresponding to the hand gesture ofthe user is not particularly limited to the type or shape thereof.

When the user input is sensed, the terminal 20 performs a controloperation corresponding to the user input. For example, when the userinput for generating the account or the user input for logging in withthe account is received (510), the terminal 20 may transmit a requestfor generation of the user account or a request for login with the useraccount to the server 10 (511). The request for the generation of theuser account or the request for the login with the user account mayinclude, for example, an ID and a password as identification informationof the user.

The server 10 may generate an account for a user or log in with apre-generated user account in response to the received request (512).Specifically, the server 10 may store the identification information ofthe user included in the request for the generation of the user accountand set and load a data directory such that data associated with thecorresponding user may be stored in association with the identificationinformation of the user. Alternatively, the server 10 may search forpre-stored identification information of the user that matches theidentification information of the user included in the request for thelogin with the user account and load the stored data directory inresponse to the retrieved identification information of the user whenthe matching identification information of the user is present.

The server 10 may transmit an account response corresponding to therequest for the generation of the user account or the request for thelogin with the user account of the terminal 20 to the terminal 20 (513).For example, the server 10 may extract information about pre-storedmodeling data or the mixed reality conference room from the loaded datadirectory. The server 10 may generate the virtual image by forming theextracted information as an object to be displayed on the mixed realityimage and transmit the generated virtual image to the terminal 20. Inthis case, the object formed in the form of the virtual image mayinclude, for example, a list of pieces of modeling data, a list of mixedreality conference rooms, or the like.

Thereafter, the server 10 and the terminal 20 may perform variousoperations on the basis of the user input for using the mixed realityservice. For example, the server 10 and the terminal 20 may performoperations such as generating, editing, deleting, managing, storing,uploading, and downloading of the three-dimensional modeling data, oroperations such as generating, editing, deleting, managing, andattending the mixed reality conference room.

Hereinafter, embodiments of various operations that may be performedthrough the mixed reality service, as described above, will be describedin more detail.

FIG. 6 is a flowchart illustrating a method of storing three-dimensionalmodeling data according to the present invention.

The method of storing the three-dimensional modeling data according tothe present invention may be performed after the above-described mixedreality service is driven. In one embodiment, the method of storing thethree-dimensional modeling data may be performed after theabove-described login with the account, but the present is not limitedthereto.

Referring to FIG. 6, the terminal 20 may receive a user input foruploading modeling data (601). The user input for uploading the modelingdata may be received through a user interface displayed as a mixedreality image.

The terminal 20 may transmit the selected modeling data to the server 10in response to the user input (602).

In one embodiment, the server 10 may convert the received modeling data(603). When the received modeling data is two-dimensional modeling data,the server 10 may convert the received modeling data intothree-dimensional modeling data in a preset format. Alternatively, whenthe received modeling data is the three-dimensional modeling data but isnot three-dimensional modeling data in a format supported by the server10 for providing the mixed reality service, the server 10 may convertthe received modeling data into three-dimensional modeling data in apreset format.

When the modeling data received from the terminal 20 is thethree-dimensional modeling data in the preset format, theabove-described conversion process may be omitted.

The server 10 may store the three-dimensional modeling data (604). Whenthe three-dimensional modeling data is generated while being logged inwith the account of the user, the server 10 may store thethree-dimensional modeling data in a data directory corresponding to thecorresponding account.

The server 10 may transmit a data upload response, which includesinformation about the three-dimensional modeling data that issuccessfully stored, to the terminal 20 (605). The response may includea virtual image including a pre-stored list of the three-dimensionalmodeling data and the like. Further, the response may further includeinformation about original modeling data of the three-dimensionalmodeling data, conversion information, information about uploaded dateand time, or the like.

In addition, the server 10 may edit, delete, manage, or download thestored three-dimensional modeling data in response to a request for editand/or management of the three-dimensional modeling data received fromthe terminal 20.

FIG. 7 is a flowchart illustrating a method of providing a mixed realityconference room according to the present invention.

Referring to FIG. 7, the terminal 20 may receive a user input forgenerating a mixed reality conference room (701). The user input forgenerating the mixed reality conference room may be received through auser interface displayed as a mixed reality image. The user input forgenerating the mixed reality conference room may include informationabout three-dimensional modeling data corresponding to the correspondingmixed reality conference room, a list of users who are allowed to attendthe corresponding mixed reality conference room, or the like. Here, thethree-dimensional modeling data may be any one of pieces ofthree-dimensional modeling data pre-stored in the server 10.

The terminal 20 may transmit a request for generation of a mixed realityconference room to the server 10 in response to the user input (702).The request for the generation of the mixed reality conference room mayinclude three-dimensional modeling data corresponding to the mixedreality conference room, a list of users who are allowed to attend themixed reality conference room, and the like, which are pieces ofinformation received through the user input.

The server 10 may generate a mixed reality conference room in responseto the request for the generation of the mixed reality conference room(703). The server 10 may assign identification information to the mixedreality conference room and map and store the three-dimensional modelingdata selected by the request. When the mixed reality conference room isgenerated while being logged in with the account of the user, the server10 may store the mixed reality conference room in a data directorycorresponding to the corresponding account.

The server 10 may transmit information about the generated mixed realityconference room to the terminal 20 in the form of a response (704). Theresponse may include a virtual image including a list of pre-generatedmixed reality conference rooms and the like. Further, the response mayfurther include identification information about the generated mixedreality conference room, information about the correspondingthree-dimensional modeling data, information about the user who isallowed to attend the mixed reality conference room, and informationabout the generated date and time.

In various embodiments, the information about the mixed realityconference room generated by the terminal 20 may be transmitted toanother terminal 20′ (706) in response to a request of the terminal 20′(705). Here, the terminal 20′ may be a terminal 20′ of a user designatedas being allowed to attend the corresponding mixed reality conferenceroom. In another embodiment, the information about the mixed realityconference room may be transmitted to the terminal 20′ by the terminal20.

Hereinafter, the terminal 20 that generates the mixed reality conferenceroom may be referred to as a master terminal or a first terminal, andthe terminal 20′ that intends to attend the mixed reality conferenceroom may be referred to as a slave terminal, a guest terminal, or asecond terminal.

When at least one mixed reality conference room is generated, at leastone of the terminals 20 and 20′ present in the mixed reality serviceproviding system 1 may receive a user input for attending the mixedreality conference room (707). In this case, the terminal 20 or 20′ maynot be the terminal 20 that generates the corresponding mixed realityconference room. In one embodiment, when the user who is allowed toattend the mixed reality conference room is designated, the terminal 20or 20′ may be the terminal 20′ of the user who is allowed to attend themixed reality conference room.

The terminal 20 or 20′ may transmit a request for attendance of themixed reality conference room, which corresponds to the user input, tothe server 10 in response to the user input (708).

The server 10 may transmit an attendance response to the attendancerequest of the terminal 20 and 20′ to the terminals 20 and 20′ (709). Invarious embodiments, the server 10 may perform device authentication,security authentication, and/or verification of whether or not to allowthe attendance of the terminals 20 and 20′ to determine whether toaccept the attendance of the mixed reality conference room of theterminals 20 and 20′, and transmit the attendance response to theterminals 20 and 20′.

When the attendance response to the mixed reality conference room isreceived, the terminals 20 and 20′ may collect tracking information inreal time (710) and transmit the collected tracking information to theserver 10 (711).

The server 10 may generate a virtual image on the basis of the receivedtracking information (712). The server 10 may generate a virtual imagefrom three-dimensional modeling data corresponding to the mixed realityconference room and generate a virtual image for a user interface objectand/or an object that imitates the user.

Specifically, the server 10 may extract space identification informationfrom the tracking information to determine a region of a specific shapein a reality space, for example, a planar region (e.g., an upper surfaceof a desk). The server 10 may generate a virtual image by arranging thethree-dimensional modeling data corresponding to the mixed realityconference room at coordinates corresponding to the planar region.

The server 10 may determine at least one of a display direction andshape of the three-dimensional modeling data on the basis of movementinformation of the terminal 20 or 20′ obtained from the trackinginformation. For example, the server 10 may rotate or move the terminal20 or 20′ according to three-dimensional rotation information ormovement information. Alternatively, the server 10 may generate avirtual image by removing or shading a portion of the three-dimensionalmodeling data to correspond to a shape of an obstacle on the basis ofthe obstacle identified from the tracking information.

Further, the server 10 may identify a position of the user and athree-dimensional rotation direction with respect to a position at whichthe three-dimensional modeling data is arranged. The server 10 mayrotate an object that imitates the user, for example, an avatar, tocorrespond to the three-dimensional rotation direction of the user, andgenerate the virtual image by arranging the object at coordinatescorresponding to the position of the user.

In various embodiments, the server 10 may generate the virtual image byassociating identification information of the corresponding user, forexample, an ID, with the generated avatar. Alternatively, in variousembodiments, when a voice chatting function or a group call function isperformed while the mixed reality conference room is open, the server 10may generate the virtual image by associating the identificationinformation of the user who is currently outputting a voice, forexample, a marker such as an arrow, with the avatar, or by associating,for example, a specific color with the avatar.

In various embodiments of the present invention, the server 10 maygenerate the virtual image by performing three-dimensional scaling.

In one example, a reality space in which the first terminal 20 islocated and a reality space in which the second terminal 20′ is locatedmay have different shapes and sizes. Further, in one example, a deskidentified by the first terminal 20 and a desk identified by the secondterminal 20′ may also have different sizes. In this case, when thethree-dimensional modeling data is displayed in a certain size in thefirst terminal 20 and the second terminal 20′, the virtual image of thethree-dimensional modeling data may not be appropriately displayed inthe terminal located in the space with a small desk (901), asillustrated in FIG. 9. Further, in this case, when an avatar of thesecond terminal 20′ is displayed at coordinates based on the realityspace of the second terminal 20′ in the first terminal 20, the avatar ofthe second terminal 20′ may not be arranged at an appropriate position(902) or may deviate from a space (903) on the mixed reality image ofthe first terminal 20, as illustrated in FIG. 9.

Therefore, the server 10 may generate the virtual image by enlarging orreducing (scaling) the three-dimensional modeling data to correspond tothe size of the identified planar region. Further, the server 10 maygenerate the virtual image by concentrating or distributing objects thatimitate users based on an arbitrary reference point according to thesize of the reality space. Here, the arbitrary reference point may becoordinates of a center of the position in which the three-dimensionalmodeling data is displayed, coordinates of a center of the recognizedplanar region, or coordinates of a center of mass according to the shapeof the recognized planar region. For example, the coordinates of theavatar for the user of the second terminal 20′ may be determined basedon the tracking information of the second terminal 20′. In this case,when the size of the reality space of the first terminal 20 is smallerthan the size of the reality space of the second terminal 20′, theserver 10 may generate the virtual image by collecting the avatar forthe user of the second terminal 20′ based on the coordinates of thecenter of the position in which the three-dimensional modeling data ofthe first terminal 20 is displayed.

Through three-dimensional scaling, a virtual image may be configured tobe appropriately displayed in a planar region in which a virtual image1001 of three-dimensional modeling data is identified even in anyterminal, as illustrated in FIG. 10, and to be displayed by arranging anavatar 1002 or 1003 of each user at an appropriate position around thethree-dimensional modeling data.

The server 10 may transmit the generated virtual image to each of theterminals 20 and 20′ (713). As described above, the virtual imagesrespectively transmitted to the terminals 20 and 20′ may be differentimages.

The terminals 20 and 20′ may synthesize the received virtual image witha reality image captured by the camera unit 21 to generate a mixedreality image (714), and display the generated mixed reality imagethrough the display unit 24 (715).

An example of the mixed reality image displayed as described above isillustrated in FIG. 11. In FIG. 11, a virtual image 1101 ofthree-dimensional modeling data is displayed on a recognized planarregion, that is, on a desk. In this case, the virtual image 1101 may bescaled to be appropriately displayed on the planar region. Further, atleast one of objects 1102 and 1103 that imitates users of one or moreother terminals is arranged to correspond to a position and rotationdirection of the corresponding user.

Further, at least one of user interface objects 1104 and 1105 may bedisplayed on some region on a screen in order to perform the controlwhich is available in the mixed reality conference room. A userinterface may include, for example, an interface for performing aprocess of editing text chatting, voice chatting, group call, memogeneration, attendee information display, attendee management, objectselection within three-dimensional modeling data, a structure, a shape,a dimension, a material, a quality of a material, a color, a pattern,and a facility of the three-dimensional modeling data.

Through the displayed mixed reality image, the user may recognize thatthe three-dimensional modeling data is displayed on the desk in thereality space and that a user of another terminal 20′ is present in thereality space. Accordingly, the user may feel a sense of realism, suchas performing a processing operation together with another user on thethree-dimensional modeling data that is really present in the realityspace in which he or she is located.

In the above-described mixed reality conference room, the server 10 andthe terminals 20 and 20′ may perform an additional operation on thethree-dimensional modeling data as in a real meeting, and share a resultthereof in real time. A description thereof will be given with referenceto FIG. 8.

Referring to FIG. 8, the terminal 20′ may receive a user input foradditional operation of the three-dimensional modeling data (801). Theuser input for additional operation may be received through thethree-dimensional modeling data and/or the user interface displayed onthe mixed reality image. The user input for additional operation mayinclude information for editing a structure, a shape, a size, amaterial, a quality of a material, a color, a pattern, and a facility ofthe three-dimensional modeling data.

The terminal 20 may transmit a request for processing of thethree-dimensional modeling data to the server 10 in response to the userinput (802).

The server 10 may edit the three-dimensional modeling data correspondingto the mixed reality conference room in response to a request forprocessing of the three-dimensional modeling data (803). The server 10may change a size, a layout, a color, a quality of a material, apattern, or the like of the object selected by the request for theprocessing in the three-dimensional modeling data, in response to therequest for the processing.

The server 10 may perform management and storage operations on thethree-dimensional modeling data, such as storing the editedthree-dimensional modeling data or storing the three-dimensionalmodeling data before editing for backup, or the like.

Thereafter, the server 10 may generate a virtual image on the basis ofthe edited three-dimensional modeling data and transmit the generatedvirtual image to the terminals 20 and 20′ (805). Each of the terminals20 and 20′ may generate a mixed reality image on the basis of thevirtual image transmitted from the server 10 (806) and display thegenerated mixed reality image (807) and thus may check and share anediting state of the three-dimensional modeling data by an arbitraryterminal 20 in real time.

FIG. 12 is a diagram illustrating a network structure of a mixed realityservice providing system according to additional embodiment of thepresent invention. In various embodiments, the mixed reality serviceproviding system illustrated in FIG. 12 may be referred to as amonitoring system, a management system, or a relay system of the mixedreality service.

Referring to FIG. 12, a mixed reality service providing system 2according to additional embodiment of the present invention may includea terminal 20 that uses the mixed reality service and a monitoringdevice 40.

In the embodiment of FIG. 12, the terminal 20 may be the same as theterminal 20 described with reference to FIGS. 1 to 11. In variousembodiments of the present invention, the terminal 20 may transmit themixed reality image displayed through the display unit 24 as amonitoring image to the external monitoring device 40. The transmissionof the mixed reality image to the monitoring device 40 may be performedby a monitoring providing unit provided in the control unit 27.

In another embodiment, the providing of the monitoring image may beperformed by the server 10. For example, the mixed reality serviceproviding unit 122 of the server 10 may transmit the mixed reality imagegenerated by the virtual image generation unit 123 as a monitoring imageto the monitoring device 40.

In still another embodiment, the providing of the monitoring image maybe performed by a monitoring server (or a relay server) providedseparately from the server 10 and the terminal 20. The monitoring servermay generate a monitoring image on the basis of at least one of thevirtual image, the reality image, and the mixed reality image which arereceived from the server 10 and/or the terminal 20 and transmit thegenerated monitoring image to the monitoring device 40.

In various embodiments, the monitoring image may be the same image asthe mixed reality image displayed through the display unit 24 of theterminal 20. Alternatively, the monitoring image may be an imageincluding the virtual image displayed through the display unit 24 of theterminal 20.

In FIG. 12, the terminal 20 is illustrated as being provided as a HMD,but a type of the terminal 20 is not limited thereto, and various typesof terminals 20 may be used as described above.

The monitoring device 40 may display the received monitoring imagethrough at least one display unit. In one embodiment, the monitoringdevice 40 may be provided as a single monitor device.

According to the present invention described above, among users in themixed reality providing system 2, a user who directly uses or does notwear the terminal 20 may visually and indirectly check a screen outputthrough the terminal 20 through the monitoring device 40. Such anembodiment may be used to manage and develop the mixed reality providingsystem 1 according to the present invention and may allow more users todirectly or indirectly attend the mixed reality conference room providedby the mixed reality providing system 1.

It will be understood by those skilled in the art that various changesin form and details may be made without departing from the spirit andscope of the present invention. In addition, the embodiments disclosedin this specification and the accompanying drawings are only examples toeasily describe the contents of the present invention and aid inunderstanding of the present invention, and the present invention is notlimited thereto. Therefore, the scope of the present invention should beinterpreted as including all the modified or deformed forms derived onthe basis of the technological scope of the present invention inaddition to the embodiments disclosed herein.

1. A method of providing a mixed reality service of a server, the methodcomprising: generating a mixed reality conference room for arbitrarythree-dimensional modeling data; receiving tracking informationcollected by at least one terminal that attends the mixed realityconference room; generating virtual images of the three-dimensionalmodeling data and at least one object on the basis of the trackinginformation; and transmitting the virtual images to the at least oneterminal, wherein the tracking information includes space identificationinformation about a reality space of each terminal and movementinformation of each terminal, the virtual images are synthesized with areality image in the at least one terminal, and the synthesized image isdisplayed as a mixed reality image, and the generating of the virtualimages includes determining at least one of a display direction and ashape of the three-dimensional modeling data on the basis of movementinformation of each terminal, determining a planar region from the spaceidentification information of each terminal, and arranging thethree-dimensional modeling data in the planar region.
 2. (canceled) 3.The method of claim 1, wherein the generating of the virtual imagesfurther includes enlarging or reducing a size of the three-dimensionalmodeling data according to a size of the planar region.
 4. The method ofclaim 1, wherein the generating of the virtual images includes:generating at least one object that imitates a user of each terminal;and determining at least one of a display direction and a position ofthe at least one object on the basis of movement information of eachterminal.
 5. The method of claim 4, wherein the determining of the atleast one of the display direction and the position of the at least oneobject further includes: determining an arbitrary reference point fromthe space identification information of each terminal; determining asize of a reality space of each terminal from the space identificationinformation of each terminal; and arranging the at least one object in aconcentrated or distributed manner based on the arbitrary referencepoint according to the size of the reality space of each terminal. 6.The method of claim 1, further comprising: receiving a request forprocessing of the three-dimensional modeling data through at least oneuser interface displayed on the mixed reality image; editing thethree-dimensional modeling data in response to the request for theprocessing; regenerating the virtual images on the basis of the editedthree-dimensional modeling data; and transmitting the regeneratedvirtual images to the at least one terminal.
 7. The method of claim 1,wherein the three-dimensional modeling data is any one of modeling datafor building design, three-dimensional data for a product, andthree-dimensional modeling data generated in a design stage.
 8. Themethod of claim 1, further comprising transmitting the mixed realityimage displayed in the terminal to a monitoring device so as to bedisplayed on the monitoring device for at least one user other than auser of the terminal.
 9. A system for providing a mixed reality service,the system comprising: at least one terminal configured to collect andtransmit tracking information while attending a mixed reality conferenceroom; and a server configured to generate a mixed reality conferenceroom for arbitrary three-dimensional modeling data, generate virtualimages of the three-dimensional modeling data and at least one object onthe basis of the tracking information received from the at least oneterminal that attends the mixed reality conference room, and transmitthe generated virtual images to the at least one terminal, wherein thetracking information includes space identification information about areality space of each terminal and movement information of eachterminal, the at least one terminal synthesizes a reality image capturedby the at least one terminal with the virtual images and displays thesynthesized image as a mixed reality image, and the server determines atleast one of a display direction and a shape of the three-dimensionalmodeling data on the basis of movement information of each terminal,determines a planar region from the space identification information ofeach terminal, and arranges the three-dimensional modeling data in theplanar region to generate the virtual image.
 10. (canceled)
 11. Thesystem of claim 9, wherein the server enlarges or reduces a size of thethree-dimensional modeling data according to a size of the planar regionand generates the virtual image.
 12. The system of claim 9, wherein theserver generates at least one object that imitates a user of eachterminal, determines at least one of a display direction and a positionof the at least one object on the basis of movement information of eachterminal, and generates the virtual image.
 13. The system of claim 12,wherein the server determines an arbitrary reference point from thespace identification information of each terminal, determines a size ofa reality space of each terminal from the space identificationinformation of each terminal, arranges the at least one object in aconcentrated or distributed manner based on the arbitrary referencepoint according to the size of the reality space of each terminal, andgenerates the virtual image.
 14. The system of claim 9, wherein: theterminal transmits a request for processing of the three-dimensionalmodeling data to the server on the basis of a user input receivedthrough at least one user interface displayed on the mixed realityimage; and the server edits the three-dimensional modeling data inresponse to the request for the processing, regenerates the virtualimages on the basis of the edited three-dimensional modeling data, andtransmits the regenerated virtual images to the at least one terminal.15. The system of claim 9, wherein the three-dimensional modeling datais any one of modeling data for building design, three-dimensional datafor a product, and three-dimensional modeling data generated in a designstage.
 16. The system of claim 9, further comprising a monitoring deviceconfigured to receive the mixed reality image displayed in the terminalfrom the terminal and display the mixed reality image for at least oneuser other than a user of the terminal.